The Impact of Textual Messages of Encouragement on Web Survey Breakoffs: An Experiment

This paper tests a new method for encouraging Web survey respondents to complete a questionnaire once they have started it and, thus, reduce the number of breakoffs. This paper builds upon prior studies that have examined the effects of using feedback mechanisms in Web surveys to encourage questionnaire completion and breakoff reduction (Crawford, Couper, & Lamias, 2001; Couper, Traugott, & Lamias, 2001; Conrad, Couper, Tourangeau, & Peytchev, 2005; Heerwegh & Loosveldt, 2006; Matzat, Snijders, & van der Horst, 2009). The proposed method displays textual messages of encouragement intermittingly to Web survey respondents. We hypothesized that such messages would reduce the number of breakoffs or, at minimum, delay eventual breakoffs. We tested this hypothesis by mounting a Web survey experiment on a national sample of college students (National Survey of Living Learning Programs, 52 institutions, N = 110,427 students). The sample was divided into 3 mutually exclusive groups that received: (1) no messages of encouragement (control), (2) brief messages of encouragement with generic content, and (3) brief messages of encouragement that contained content tailored to their specific school. The results suggest no association between displaying messages of encouragement and deterring or delaying breakoffs. Further, no association was found between messages of encouragement and the amount of time respondents spent logged in to the survey instrument or the number of items presented to them. The failure to find evidence is analyzed and detailed recommendations for further research on the relationship between feedback mechanisms and survey completion are given.